<!DOCTYPE html>
<html lang=zh-CN>
<head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="google-site-verification" content="xBT4GhYoi5qRD5tr338pgPM5OWHHIDR6mNg1a3euekI" />
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta property="og:description" content="samuel-24276的博客">
    <meta property="og:type" content="website">
    <meta name="description" content="samuel-24276的博客">
    <meta name="keyword"  content="aircloud">
    <link rel="shortcut icon" href="/myblog/img/favicon.ico">

    <title>
        
        学习爬虫过程中的一些坑 - samuel-24276的博客 | samuel-24276&#39;s Blog
        
    </title>

    <!-- Custom CSS -->
    
<link rel="stylesheet" href="/myblog/css/aircloud.css">

    
<link rel="stylesheet" href="/myblog/css/gitment.css">

    <!--<link rel="stylesheet" href="https://imsun.github.io/gitment/style/default.css">-->
    <link href="//at.alicdn.com/t/font_620856_28hi1hpxx24.css" rel="stylesheet" type="text/css">
    <!-- ga & ba script hoook -->
    <script></script>
<meta name="generator" content="Hexo 7.1.1"></head>

<body>

<div class="site-nav-toggle" id="site-nav-toggle">
    <button>
        <span class="btn-bar"></span>
        <span class="btn-bar"></span>
        <span class="btn-bar"></span>
    </button>
</div>

<div class="index-about">
    <i> Like what you do, do what you like </i>
</div>

<div class="index-container">
    
    <div class="index-left">
        
<div class="nav" id="nav">
    <div class="avatar-name">
        <div class="avatar ">
            <img src="/myblog/img/avatar.jpg" />
        </div>
        <div class="name">
            <i>samuel-24276</i>
        </div>
    </div>
    <div class="contents" id="nav-content">
        <ul>
            <li >
                <a href="/myblog/">
                    <i class="iconfont icon-shouye1"></i>
                    <span>主页</span>
                </a>
            </li>
            <li >
                <a href="/myblog/tags">
                    <i class="iconfont icon-biaoqian1"></i>
                    <span>标签</span>
                </a>
            </li>
            <li >
                <a href="/myblog/archives">
                    <i class="iconfont icon-guidang2"></i>
                    <span>存档</span>
                </a>
            </li>
            <li >
                <a href="/myblog/collect/">
                    <i class="iconfont icon-shoucang1"></i>
                    <span>收藏</span>
                </a>
            </li>
            <li >
                <a href="/myblog/about/">
                    <i class="iconfont icon-guanyu2"></i>
                    <span>关于</span>
                </a>
            </li>
            
            <li>
                <a id="search">
                    <i class="iconfont icon-sousuo1"></i>
                    <span>搜索</span>
                </a>
            </li>
            
        </ul>
    </div>
    
        <div id="toc" class="toc-article">
    <ol class="toc"><li class="toc-item toc-level-1"><a class="toc-link" href="#1-Python%E8%AF%AD%E6%B3%95%E5%8F%8APycharm"><span class="toc-text">1.Python语法及Pycharm</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#1-1-%E6%B3%A8%E9%87%8A"><span class="toc-text">1.1.注释</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#1-2-%E7%A9%BA%E6%A0%BC"><span class="toc-text">1.2.空格</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#1-3-Pycharm"><span class="toc-text">1.3.Pycharm</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#2-%E7%88%AC%E8%99%AB"><span class="toc-text">2.爬虫</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#2-1-Python3-7%E4%BD%BF%E7%94%A8etree-%E4%B8%8Expath%E6%90%AD%E9%85%8D%E4%BD%BF%E7%94%A8"><span class="toc-text">2.1.Python3.7使用etree(与xpath搭配使用)</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#2-2-pandas%E7%94%A8%E4%BA%8E%E8%A1%A8%E6%A0%BC%E5%8C%96%E6%95%B0%E6%8D%AE"><span class="toc-text">2.2.pandas用于表格化数据</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#2-3-%E6%96%87%E4%BB%B6%E6%93%8D%E4%BD%9C%E4%B8%8Eopen"><span class="toc-text">2.3.文件操作与open()</span></a><ol class="toc-child"><li class="toc-item toc-level-3"><a class="toc-link" href="#2-3-1-open-%E7%9A%84%E6%A8%A1%E5%BC%8F%E9%80%89%E6%8B%A9"><span class="toc-text">2.3.1.open()的模式选择</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#2-3-2-file-%E5%AF%B9%E8%B1%A1%E6%96%B9%E6%B3%95"><span class="toc-text">2.3.2.file 对象方法</span></a></li></ol></li><li class="toc-item toc-level-2"><a class="toc-link" href="#2-4-%E5%BC%82%E5%B8%B8%E6%83%85%E5%86%B5"><span class="toc-text">2.4.异常情况</span></a></li></ol></li></ol>
</div>
    
</div>


<div class="search-field" id="search-field">
    <div class="search-bg" id="search-bg"></div>
    <div class="search-container">
        <div class="search-input">
            <span id="esc-search"> <i class="icon-fanhui iconfont"></i></span>
            <input id="search-input"/>
            <span id="begin-search">搜索</span>
        </div>
        <div class="search-result-container" id="search-result-container">

        </div>
    </div>
</div>

        <div class="index-about-mobile">
            <i> Like what you do, do what you like </i>
        </div>
    </div>
    
    <div class="index-middle">
        <!-- Main Content -->
        


<div class="post-container">
    <div class="post-title">
        学习爬虫过程中的一些坑
    </div>

    <div class="post-meta">
        <span class="attr">发布于：<span>2020-03-30 22:14:38</span></span>
        
        <span class="attr">标签：/
        
        <a class="tag" href="/myblog/tags/#爬虫" title="爬虫">爬虫</a>
        <span>/</span>
        
        <a class="tag" href="/myblog/tags/#Python" title="Python">Python</a>
        <span>/</span>
        
        
        </span>
        <span class="attr">访问：<span id="busuanzi_value_page_pv"></span>
</span>
</span>
    </div>
    <div class="post-content no-indent">
        <h1 id="1-Python语法及Pycharm"><a href="#1-Python语法及Pycharm" class="headerlink" title="1.Python语法及Pycharm"></a>1.Python语法及Pycharm</h1><h2 id="1-1-注释"><a href="#1-1-注释" class="headerlink" title="1.1.注释"></a>1.1.注释</h2><p>单行注释： <code>#</code> 后跟一个空格，否则出现警告</p>
<p>多行注释： <code>&#39;&#39;&#39;</code> 是单引号，不是markdown代码块的三个反引号</p>
<h2 id="1-2-空格"><a href="#1-2-空格" class="headerlink" title="1.2.空格"></a>1.2.空格</h2><p>函数之间用两行空格间隔，否则出现警告</p>
<p>给数组&#x2F;元组赋值时，若值为键值对，则键加单引号，后紧跟冒号，冒号后面留一个空格，跟值，否则出现警告</p>
<h2 id="1-3-Pycharm"><a href="#1-3-Pycharm" class="headerlink" title="1.3.Pycharm"></a>1.3.Pycharm</h2><p>打开后写任何代码都有红色错误：File-&gt;Settings-&gt;Project-&gt;Interpreter,选择安装路径的Python.exe而不是当前目录下的Python.exe</p>
<h1 id="2-爬虫"><a href="#2-爬虫" class="headerlink" title="2.爬虫"></a>2.爬虫</h1><h2 id="2-1-Python3-7使用etree-与xpath搭配使用"><a href="#2-1-Python3-7使用etree-与xpath搭配使用" class="headerlink" title="2.1.Python3.7使用etree(与xpath搭配使用)"></a>2.1.Python3.7使用etree(与xpath搭配使用)</h2><figure class="highlight abnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">import lxml.html</span><br><span class="line"><span class="attribute">etree</span> <span class="operator">=</span> lxml.html.etree</span><br></pre></td></tr></table></figure>

<h2 id="2-2-pandas用于表格化数据"><a href="#2-2-pandas用于表格化数据" class="headerlink" title="2.2.pandas用于表格化数据"></a>2.2.pandas用于表格化数据</h2><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line">变量<span class="number">2</span> = pd.DataFrame(未处理的变量<span class="number">1</span>)</span><br><span class="line">变量<span class="number">2.</span>to_csv(<span class="string">&#x27;文件名.csv&#x27;</span>, encoding=<span class="string">&#x27;utf_8_sig&#x27;</span>)</span><br></pre></td></tr></table></figure>

<h2 id="2-3-文件操作与open"><a href="#2-3-文件操作与open" class="headerlink" title="2.3.文件操作与open()"></a>2.3.文件操作与open()</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">if</span> <span class="string">&#x27;存储目录名&#x27;</span> <span class="keyword">in</span> os.listdir(<span class="string">r&#x27;存储目录名的绝对路径&#x27;</span>):</span><br><span class="line">	<span class="keyword">pass</span></span><br><span class="line"><span class="keyword">else</span>:</span><br><span class="line">	os.mkdir(<span class="string">&#x27;存储目录名&#x27;</span>)</span><br><span class="line">os.chdir(<span class="string">r&#x27;存储目录名的绝对路径&#x27;</span>)</span><br></pre></td></tr></table></figure>

<p>以上代码用于检查存储目录是否存在，不存在则创建，<br>注意：os.listdir()及os.mkdir()都是有参数r的，不要遗忘</p>
<figure class="highlight stylus"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="selector-tag">img</span> = requests<span class="selector-class">.get</span>(url)<span class="selector-class">.content</span></span><br><span class="line">with <span class="built_in">open</span>(movie<span class="selector-attr">[<span class="string">&#x27;name&#x27;</span>]</span> + <span class="string">&#x27;.jpg&#x27;</span>, <span class="string">&#x27;wb+&#x27;</span>) as f:</span><br><span class="line">	f<span class="selector-class">.write</span>(img)</span><br></pre></td></tr></table></figure>

<p>以上代码用于存储图片，首先确定图片路径url，在图片上右击，检查，定位到的src即为路径url，第一行代码获取图片的二进制内容。open()常用的有两个参数，para1为文件名字，Para2为模式</p>
<h3 id="2-3-1-open-的模式选择"><a href="#2-3-1-open-的模式选择" class="headerlink" title="2.3.1.open()的模式选择"></a>2.3.1.open()的模式选择</h3><table>
<thead>
<tr>
<th>模式</th>
<th align="center">描述</th>
</tr>
</thead>
<tbody><tr>
<td>r</td>
<td align="center">以只读方式打开文件。文件的指针将会放在文件的开头。这是默认模式。</td>
</tr>
<tr>
<td>rb</td>
<td align="center">以二进制格式打开一个文件用于只读。文件指针将会放在文件的开头。这是默认模式。</td>
</tr>
<tr>
<td>r+</td>
<td align="center">打开一个文件用于读写。文件指针将会放在文件的开头。</td>
</tr>
<tr>
<td>rb+</td>
<td align="center">以二进制格式打开一个文件用于读写。文件指针将会放在文件的开头。</td>
</tr>
<tr>
<td>w</td>
<td align="center">打开一个文件只用于写入。如果该文件已存在则打开文件，并从开头开始编辑，即原有内容会被删除。如果该文件不存在，创建新文件。</td>
</tr>
<tr>
<td>wb</td>
<td align="center">以二进制格式打开一个文件只用于写入。如果该文件已存在则打开文件，并从开头开始编辑，即原有内容会被删除。如果该文件不存在，创建新文件。</td>
</tr>
<tr>
<td>w+</td>
<td align="center">打开一个文件用于读写。如果该文件已存在则打开文件，并从开头开始编辑，即原有内容会被删除。如果该文件不存在，创建新文件。</td>
</tr>
<tr>
<td>wb+</td>
<td align="center">以二进制格式打开一个文件用于读写。如果该文件已存在则打开文件，并从开头开始编辑，即原有内容会被删除。如果该文件不存在，创建新文件。</td>
</tr>
<tr>
<td>a</td>
<td align="center">打开一个文件用于追加。如果该文件已存在，文件指针将会放在文件的结尾。也就是说，新的内容将会被写入到已有内容之后。如果该文件不存在，创建新文件进行写入。</td>
</tr>
<tr>
<td>ab</td>
<td align="center">以二进制格式打开一个文件用于追加。如果该文件已存在，文件指针将会放在文件的结尾。也就是说，新的内容将会被写入到已有内容之后。如果该文件不存在，创建新文件进行写入。</td>
</tr>
<tr>
<td>a+</td>
<td align="center">打开一个文件用于读写。如果该文件已存在，文件指针将会放在文件的结尾。文件打开时会是追加模式。如果该文件不存在，创建新文件用于读写。</td>
</tr>
<tr>
<td>ab+</td>
<td align="center">以二进制格式打开一个文件用于追加。如果该文件已存在，文件指针将会放在文件的结尾。如果该文件不存在，创建新文件用于读写。</td>
</tr>
</tbody></table>
<h3 id="2-3-2-file-对象方法"><a href="#2-3-2-file-对象方法" class="headerlink" title="2.3.2.file 对象方法"></a>2.3.2.file 对象方法</h3><ul>
<li><p>file.read([size])：size 未指定则返回整个文件，如果文件大小 &gt;2 倍内存则有问题，f.read()读到文件尾时返回””(空字串)。</p>
</li>
<li><p>file.readline()：返回一行。</p>
</li>
<li><p>file.readlines([size]) ：返回包含size行的列表, size 未指定则返回全部行。</p>
</li>
<li><p>for line in f: print line ：通过迭代器访问。</p>
</li>
<li><p>f.write(“hello\n”)：如果要写入字符串以外的数据,先将他转换为字符串。</p>
</li>
<li><p>f.tell()：返回一个整数,表示当前文件指针的位置(就是到文件头的比特数)。</p>
</li>
<li><p>f.close() 关闭文件</p>
</li>
</ul>
<h2 id="2-4-异常情况"><a href="#2-4-异常情况" class="headerlink" title="2.4.异常情况"></a>2.4.异常情况</h2><ul>
<li>网页爬取时执行状态成功，但获取不到想要的数据<br>当我们爬取网页的内容，对源代码进行请求，响应的源代码中没有我们需要的东西时，需要查看 <code>Network</code> ,其功能是：记录在当前页面上所发生的所有请求（它是实时加载的，如果是空的，则需要刷新网页）。</li>
</ul>
<p>F12-&gt;F5-&gt;Network-&gt;勾选XHR-&gt;点击网页上想要获取的信息，此时会新弹出来一个请求，左击该请求，并左击状态栏的Headers，选中下面的<code>Request URL</code> 作为请求的url即可。</p>
<hr>

        
            <div class="donate-container">
    <div class="donate-button">
        <button id="donate-button">赞赏</button>
    </div>
    <div class="donate-img-container hide" id="donate-img-container">
        <img id="donate-img" src="" data-src="/myblog/img/donate.jpg">
        <p> 感谢鼓励 </p>
    </div>
</div>
        
        <br />
        <div id="comment-container">
        </div>
        <div id="disqus_thread"></div>
        <div id="lv-container"></div>
        <div class="giscus"></div>
    </div>
</div>

    </div>
</div>


<footer class="footer">
    <ul class="list-inline text-center">
        
        

        

        

        
        <li>
            <a target="_blank"  href="https://github.com/samuel-24276">
                            <span class="fa-stack fa-lg">
                                <i class="iconfont icon-github"></i>
                            </span>
            </a>
        </li>
        

        

    </ul>
    
    <p>
        <span>/</span>
        
        <span><a target="_blank" rel="noopener" href="https://github.com/samuel-24276">samuel-24276&#39;s Page</a></span>
        <span>/</span>
        
        <span><a target="_blank" rel="noopener" href="https://beian.miit.gov.cn/">北京-未备案</a></span>
        <span>/</span>
        
        <span><a href="#">返回顶部</a></span>
        <span>/</span>
        
    </p>
    
    <p>
        <span id="busuanzi_container_site_pv">
            <span id="busuanzi_value_site_pv"></span>PV
        </span>
        <span id="busuanzi_container_site_uv">
            <span id="busuanzi_value_site_uv"></span>UV
        </span>
        Created By <a target="_blank" rel="noopener" href="https://hexo.io/">Hexo</a>  Theme <a target="_blank" rel="noopener" href="https://github.com/aircloud/hexo-theme-aircloud">AirCloud</a></p>
</footer>




</body>

<script>
    // We expose some of the variables needed by the front end
    window.hexo_search_path = "search.json"
    window.hexo_root = "/myblog/"
    window.isPost = true
</script>
<script src="https://cdn.bootcss.com/jquery/3.3.1/jquery.min.js"></script>

<script src="/myblog/js/index.js"></script>

<script async src="//busuanzi.ibruce.info/busuanzi/2.3/busuanzi.pure.mini.js"></script>






    <script type="text/javascript">
       (function() {
           if (typeof LivereTower === 'function') { return; }

           var j, d = document.getElementById('lv-container');

           d.setAttribute('data-id','city');
           d.setAttribute('data-uid' , 'MTAyMC80OTA5OC8yNTU5Mw==');

           j = document.createElement('script');
           j.src = 'https://cdn-city.livere.com/js/embed.dist.js';
           j.async = true;

           d.appendChild(j);
       })();
    </script>
    <noscript>为正常使用来必力评论功能请激活JavaScript</noscript>
    </div>

</html>
